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1)  Background 

Some  studies  [1,  2]  suggest  that  the  pattern  of  a  subject’s  eye  movements  may  be 
one  indicator  of  whether  a  subject  is  lying  or  telling  the  truth.  Prior  to  testing  this 
hypothesis,  it  is  important  to  develop  a  real-time,  non-invasive  automatic  eye  tracking 
system  to  detect  and  track  the  eye  movements  of  a  subject. 

Currently,  there  are  medical  systems  (video  oculography  systems)  available  for 
patient  eye  movement  tracking.  However,  these  systems  are  not  suitable  for  our 
application.  To  limit  the  head  movement,  the  video  oculography  systems  typically  require 
that  the  subject  1)  wear  a  head  mounted  camera  system,  2)  restrict  his  gaze  to  a  very 
limited  space  in  front  of  him,  or  3)  use  a  chin  rest  to  support  his  head.  For  a  survey  of 
oculography  systems  refer  to  [3]. 

In  contrast  to  the  medical  oculography  systems,  our  approach  allows  the  subject  a 
much  greater  degree  of  freedom  to  move  and  look  around.  While  sitting  about  eight  feet 
in  front  of  the  digital  camera,  subjects  are  allowed  to  move  their  heads  freely  and  to  gaze 
anywhere  they  wish.  This  type  of  setup  is  very  difficult  to  achieve  with  a  medical  video 
oculography  system.  There  are  many  factors  that  hinder  the  successful  detection  and 
tracking  of  eye-movement,  for  example: 

•  The  head  movement  of  a  subject  -  when  the  subject  moves  his/her  head  quickly,  it 
can  create  a  relatively  high  speed  eye  movement  which  can  blur  the  pupil  image 

•  The  oblique  look  directions  of  the  subject  -  when  the  subject  looks  in  an  oblique 
direction,  the  pupils  will  not  reflect  the  infrared  (IR)  illumination  directly  back 
into  the  camera,  thus  reducing  the  brightness  of  the  pupil  images 

•  The  subject’s  eyeglasses  can  sometimes  block  the  reflection  from  the  pupils, 
resulting  in  a  failure  to  detect  the  eyes 

•  Jewelry  and  metal  framed  eye-glasses  can  reflect  strong  external  illumination  and 
cause  a  false  detection  of  the  pupils 

•  Different  eye  colors  can  cause  variable  contrast  in  the  iris  area,  and  these 
variations  can  affect  the  threshold  setting  and  cause  detection  failure 

The  analysis  that  follows  was  conducted  after  the  fact,  using  data  collected  at  another 
facility  using  a  simple  experimental  set  up.  The  subjects  were  photographed  with  a  single 
camera  equipped  with  a  single  IR  illuminator.  The  IR  illumination  was  constant  for  each 
frame.  Alternative  approaches  to  addressing  the  above  problems  exist,  but  require  more 
sophisticated  systems  for  the  initial  data  collection.  For  example,  according  to  Zhu  and  Ji 
[4],  the  most  effective  way  to  remove  false  detections  due  to  background  clutter 
(eyeglasses  and  jewelry)  is  to  compute  the  image  difference  between  two  alternate  IR 
illuminations:  one  that  is  on-axis  and  one  that  is  off-axis.  These  consecutive  images  are 
captured  in  alternate  camera  frames.  The  on-axis  IR  illumination  produces  a  strong 
reflection  from  the  pupils,  while  the  off-axis  IR  illumination  results  in  a  weak  reflection. 
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Assuming  that  the  background  does  not  change  dramatically  between  these  two  IR 
illuminated  images,  computing  the  image  difference  will  remove  some  of  the  false  pupil 
detections  caused  by  background  variation. 

Since  our  goal  is  to  achieve  real-time  detection,  algorithms  that  are  too 
complicated  or  require  a  long  processing  time  are  unsuitable.  Similarly,  due  to  the  limited 
number  of  available  subjects,  the  training  based  pattern  recognition  methods  for  eye 
detection  are  also  inapplicable.  In  the  following  sections,  the  algorithm  that  we  developed 
for  detecting  and  tracking  the  pupils  is  described. 

2)  Approach 

Figure  1  shows  the  data  flow  diagram  of  the  new  pupil  detection  and  tracking 
algorithm.  First,  we  read  in  an  infrared  image,  as  shown  in  Figure  2.  Then  the  infrared 
image  is  converted  into  a  gray  scale  image,  as  shown  in  Figure  3.  Most  morphological 
filters  require  gray  scale  imagery  data. 


Read  in 
an  IR  image 


Convert  it 
to  a  gray 
scale 
image 


Threshold  the 
gray  scale 
image  into  a 
binary  image 


Apply  a  morphological 
opening  filter  on  the 
binary  image  to  remove 
any  small  white  clutter 
in  the  binary  image 


Apply  a 
morphological 
closing  filter  on  the 
previously  filtered 
binary  image  to  fill 
in  any  small  gaps  in 
the  image 


Discard  ellipse  that  has 

1 )  either  a  too  low  average 
pixel  value 

2)  or  a  too  high  average 
pixel  value 

in  the  associated  iris  area 


Select  the  two  largest  ellipses 

if  there  are  more  than  two  in  the  image 


Figure  1  -  Data  flow  diagram  of  pupil  detection 
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Original  IR  image 


Figure  2  -  An  original  infrared  image. 

Gray  scale  version 


Figure  3  -  The  gray  version  of  the  original  image. 
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a)  Thresholding 

The  gray  scale  image  is  thresholded  with  a  properly  selected  threshold  value.  The 
selected  threshold  value  is  204  (80%  of  the  white  level).  The  resultant  binary  image  of 
the  thresholding  process  is  shown  in  Figure  4.  This  figure  shows  several  bright  areas 
above  the  threshold  value:  two  pupils,  one  earlobe,  the  chin,  two  bright  patches  between 
the  earlobe  and  the  right  pupil,  the  white  shirt  and  some  bright  patches  of  background. 

It  is  well  established  in  image  processing  that  too  low  a  threshold  value  may  generate 
too  many  false  detections  while  too  high  a  threshold  value  may  prevent  some  true 
(correct)  detections.  Properly  choosing  a  viable  threshold  value  is  a  key  issue.  There  are 
different  threshold  algorithms  [5].  Some  are  based  on  clustering  criteria,  some  on 
histogram  equalization,  some  on  image  entropy,  and  some  on  object  attributes.  A 
threshold  method  based  on  a  computationally-inexpensive  histogram  equalization  was 
tested,  but  this  method  did  not  seem  to  be  very  effective.  Therefore,  a  simple  hard- 
threshold  method  is  adopted  here.  Due  to  the  varying  lighting  conditions  for  different 
subjects,  the  latter  method  does  not  always  perform  as  well  as  desired.  The  best  working 
threshold  value  for  individual  subject  and  lighting  condition  was  selected  by  trial  and 
error.  Further  investigation  is  needed  to  identify  which  method  will  be  the  best  for 
automatic  image  thresholding,  given  variable  lighting  conditions  across  subjects. 


Figure  4  -  A  resultant  binary  image  after  the  thresholding. 
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b)  Morphological  filtering 

We  will  next  apply  morphological  filters  [6]  to  the  thresholded  binary  image.  There 
are  three  consecutive  stages  of  morphological  filtering:  1)  opening,  2)  closing,  and  3) 
boundary  detection. 

We  use  the  morphological  opening  filter  to  remove  any  small  objects  from  an  image, 
while  preserving  the  shape  and  size  of  the  larger  objects  present.  In  essence,  the 
morphological  opening  is  used  to  remove  small  white  dots  in  the  image.  A  resultant 
image  is  shown  in  Figure  5.  Comparing  this  figure  with  Figure  4,  notice  that  most  of  the 
small  white  dots  are  removed. 

We  then  apply  the  morphological  closing  filter  to  fill  small  gaps  in  the  larger  objects 
in  the  image,  while  preserving  the  shape  and  size  of  these  objects.  In  essence,  the 
morphological  closing  filter  is  used  to  fill  the  small  black  spots  in  the  image  with  white 
pixel  values.  This  procedure  also  smoothes  the  edges  of  the  large  objects  in  the  image.  A 
resultant  image  is  shown  in  Figure  6.  Comparing  this  figure  with  Figure  5,  notice  that 
many  of  the  small  black  holes  are  filled  with  white  pixel  values. 

In  the  third  step  of  morphological  filtering,  we  apply  the  boundary  detector  to  trace 
the  exterior  boundaries  of  objects,  as  well  as  the  boundaries  of  the  holes  inside  of  these 
objects,  within  the  binary  image.  A  resultant  image  is  shown  in  Figure  7.  In  this  image, 
the  detected  boundaries  are  overlaid  on  the  original  infrared  image.  There  are  eight 
objects  in  total:  two  pupils,  one  earlobe,  two  small  white  patches  between  the  earlobe  and 
the  right  eye,  the  chin,  the  white  shirt  merged  with  a  bright  background  patch  just  above 
the  right  shoulder  and  the  bright  background  patch  just  above  the  left  shoulder. 

All  of  the  morphological  filter  operations  are  associated  with  a  ’structure  element’ 
that  defines  the  size  of  an  operation  kernel,  and  different  sizes  of  the  structure  element 
can  produce  slightly  different  operational  results.  Some  trial  and  error  is  required  to  find 
the  proper  size  for  the  structure  element. 
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Figure  5  -  A  resultant  image  after  the  morphological  opening  process. 


Figure  6  -  A  resultant  image  after  the  morphological  closing  process. 
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Boundary  drawn 


Figure  7  -  The  detected  boundaries  overlaid  on  the  original  infrared  image. 


c)  Fitting  the  detected  objects  with  ellipses 

After  detecting  the  constituent  objects  using  morphological  filtering,  it  is  necessary  to 
fit  the  objects  with  ellipses.  Some  researchers  [7,  8]  propose  fitting  the  detected  objects 
with  circles,  because  the  pupil  itself  is  circular.  Alternatively,  Z.  Zhu  and  Q.  Ji  [9] 
proposed  using  ellipses  to  fit  the  detected  objects. 

Initially,  we  fitted  the  detected  objects  with  circles.  We  did  achieve  some  good 
results,  but  circular  fitting  always  failed  when  the  subject  presented  with  head  motion. 
After  further  investigation,  we  concluded  that  fitting  the  detected  pupils  with  a  circle 
works  only  when  there  is  no  head  motion  by  the  subject.  When  the  subject  does  initiate 
head  movement,  the  pupil  will  appear  as  a  streak,  rather  than  as  a  circle,  due  to  the  high¬ 
speed  movement  of  the  eye  relative  to  the  digital  sampling  speed  of  the  camera.  Fitting 
the  streak  of  a  pupil  with  a  circle  often  resulted  in  failure  to  detect  the  pupil  at  all.  In 
Figure  8  a  circle  was  successfully  fitted  to  one  elongated  pupil,  but  not  to  the  other. 
Figure  9  shows  two  ellipses  successfully  fitted  to  both  elongated  pupils. 
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Figure  8  -Fitting  the  elongated  pupils  with  circles  often  failed. 


Figure  9  -  Fitting  the  elongated  pupils  with  ellipses  had  a  higher  successful  rate  of  pupil  detection  rate. 
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After  recognizing  this  sampling  issue,  the  circle  fitting  was  replaced  with  ellipse 
fitting.  This  modified  algorithm  improves  the  pupil  detection  quite  dramatically.  Figure 
10  shows  the  detected  objects  with  fitted  ellipses.  The  size  of  ellipse  can  affect  the 
outcome.  The  maximum  size  of  the  ellipse  is  set  to  be  255  pixels  in  this  example; 
therefore,  any  object  that  has  a  greater  size  than  255  pixels  is  removed.  In  Figure  7,  the 
largest  white  area  is  the  white  shirt  merged  with  a  bright  spot  just  above  the  right 
shoulder  of  the  subject.  This  artifact  is  eliminated  in  Figure  10.  By  applying  the 
“constraint  of  ellipse  size“,  unwanted  ellipses  may  be  removed. 


Gray  scale  Image  with  detected  RO  s  (constrained  size=255) 


Figure  10  -  Detected  ellipses  overlaid  on  the  gray  scale  image. 


We  can  also  reduce  the  size  of  the  ellipse  to  approximate  the  size  of  pupil  in  the 
image.  This  will  eliminate  detection  of  the  earlobe,  the  chin  and  the  bright  spot  just 
above  the  left  shoulder.  The  results  of  fitting  these  objects  with  a  smaller  ellipse  are 
shown  in  Figure  11.  Here,  the  size  of  ellipse  is  further  reduced  to  14  pixels. 


d)  Add  image  intensity  constraints  on  the  detected  ellipse 

With  a  smaller  size  ellipse  (i.e.,  no  larger  than  14  pixels),  we  can  eliminate  the 
false  detection  due  to  the  earlobe,  the  chin,  and  the  bright  background  patch  just  above 
the  left  shoulder  as  shown  in  Figure  9.  Nonetheless,  there  are  some  false  detections 
remaining;  for  example,  the  two  bright  spots  between  the  earlobe  and  the  right  eye.  The 
next  question  is  -  how  can  we  eliminate  false  detections  that  satisfy  the  same  size  criteria 
as  the  real  pupil?  The  answer  lies  in  applying  the  “constraints  of  local  image  intensity”. 
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We  observe  that  the  associated  iris  area  is  usually  darker  than  the  pupil,  while  a 
bright  spot  that  is  a  false  detection  does  not  have  a  comparable  dark  area  adjacent  to  it.  A 
lower  half  circle  with  a  diameter  proportional  to  the  pupil  size  is  use  to  model  the  iris 
area  adjacent  to  a  potential  pupil.  The  green  half  circles  in  Figure  1 1  shows  the  postulated 
iris  area  superimposed  on  the  candidate  pupil  detections. 

The  average  pixel  value  of  the  pupil  and  its  associated  iris  area  are  calculated  for 
each  potential  pupil.  These  two  pixel  average  values  are  compared  against  their 
corresponding  threshold  value.  A  potential  pupil  identification  will  be  discarded  if  the 
average  pixel  value  of  the  pupil  is  too  low  or  if  the  average  pixel  value  of  the  associated 
iris  area  is  too  high.  This  local  image  intensity  thresholding  method  is  an  application  of 
the  “constraints  of  local  image  intensity”. 

It  is  observed  that  different  subjects  may  have  different  average  pixel  values  for 
their  iris  area  due  to  different  eye  colors.  Therefore,  the  threshold  value  for  the  average 
pupil  and  the  iris  area  is  obtained  by  trial  and  error.  The  final  detection  result  is  shown  in 
Figure  12.  Here,  the  detected  pupils  are  overlaid  on  the  original  infrared  image. 


Gray  sca  e  image  with  cetected  ROIs  (Constrained  size=14} 


Figure  11  -  The  detected  small  ellipse  with  associated  iris  area. 
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Fira  resu  t 


Figure  12  -  The  final  detected  pupils  are  overlaid  on  the  original  IR  image. 


3)  Conclusion  and  Discussion 

We  have  demonstrated  the  success  of  a  morphological  filtering  approach  on  pupil 
detection.  There  are  some  issues  however,  requiring  further  investigation  before  a  fully 
automatic  system  can  be  realized.  Specifically,  the  selection  of  different  threshold  values 
is  based  on  trial  and  error.  We  would  need  to  develop  automatic  approaches  for 
normalizing:  1)  the  image  intensity,  2)  the  average  size  of  human  pupil,  and  3)  the 
average  pixel  value  for  an  iris  area  with  different  eye  colors.  In  addition,  it  is  possible  that 
false  detections  may  pass  through  the  “ellipse  size  constraint”  and  the  “local  image 
intensity  constraint”  described  here.  Removing  these  false  detections  remains  a 
challenging  problem. 

Regarding  data  collection,  the  errors  in  the  pupil  detection  process  may  be  aided 
by  consistently  using  the  same  geometry  of  subject,  lighting  source,  and  camera,  as  well 
as  the  same  lighting  intensity.  Additionally,  control  of  reflections  that  may  cause  false 
alarms  is  also  desirable,  perhaps  by  having  jewelry  removed  and  darker  clothing  worn  by 
subjects. 
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